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Introduction 



* This is a course on SEE Test Plan 
development 

* It is NOT 

- How to test or testing methodology 

- A detailed discussion of technology 

- New material on new effects 

* It is 

- An introductory discussion of the items that go into 
planning an SEE test that should complement the 
SEE test methodology used 

* Material will only cover heavy ion SEE testing 
and not proton, LASER, or other though many 
of the discussed items may be applicable. 
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Course Abstract 



* While standards and guidelines for how-to 
perform single event effects (SEE) testing have 
existed almost since the first cyclotron testing, 
guidance on the development of SEE test plans 
has not been as easy to find. 

* In this section of the short course, we attempt to 
rectify this lack. 

* We consider the approach outlined here as a 
“living” document: 

- mission specific constraints and new technology related 
issues always need to be taken into account. 

* We note that we will use the term “test planning” 
in the context of those items being included in a 
test plan. 


SEE Test Planning 101, Seville, SP - LaBel, Pellish, Berg Sep 19 2011 


4 


Requirements - 
Dual and Competing Nature(s) 



• Programmatic 

- Cost 

- Schedule 

- Personnel 

- Availability 

- Criticality 

- RISK! 


• Technical 

- Device 

- Packaging 

- Beam/facility 

- Application 

- Data Capture 


Dual Nature 2: Flight Project versus Research 

How we plan and prepare for a test will also vary 

with this trade space 

All tests are driven by requirements and objectives in 

one manner or another 


SEE Test Planning 101, Seville, SP - LaBel, Pellish, Berg Sep 19 2011 


5 


Flight Project Requirements 

When planning a test for a flight project, considerations 
may include: 

- Acceptance criteria 

• Error or fail rate (System or Device) 

- System availability may be appropriate, as well 

• Minimum device hardness level 

- Linear Energy Transfer threshold (LETth), for example 

• Error definition and application information 

- User application(s) 

• Circuit 

- We note that “test as you fly” is recommended 

• Criticality 

- Programmatic constraints 

The bottom line is that flight project tests are usually 
application specific and designed to get a specific answer 
such as: 

- Is the SEL threshold higher than X? or 

- Will I see an effect more than once every 10 days? 
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Research Requirements 



* These are less specific than requirements for 
flight projects and may include 

- Generic technology/device hardness 

- Application range 

- Angular exploration 

- Frequency exploration 

- Beam characteristics such as ion/energy/range effects 

- Error propagation, charge sharing, etc... 

- Programmatic constraints 

* The bottom line is that all requirements and 
objectives should be “in plan”, i.e., considered 
prior to test and included in test plan 
development. 
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Resource Estimation 



* Many factors will weigh in to actual resource (re: 
cost and schedule) considerations including: 

- Complexity of device/test and preparation thereof 

- Facility availability (and time allotment) 

- Urgency of test 

- Funds availability, and so forth 

* We usually try to “pre-plan” facility access 
approximately three months prior to a test date 
and refine the list as flight project exigencies, test 
readiness levels, etc are evaluated. 

- At NASA, flight projects receive priority in planning 

* Schedules should be developed and included that 
include all phases of testing from requirements 
definition to completed report. 


SEE Test Planning 101, Seville, SP - LaBel, Pellish, Berg Sep 19 2011 


8 


Cost Estimation Factors 



• Labor 

- Principal investigator/team lead 

- Test engineers/technicians 

• Electrical, mechanical, VHDL, software, cabling, etc. 

- Test performance (pay attention to overtime needs) 

- Data Analysis 

- Report and plan writing 

• Non-recurring engineering costs 

• Board fabrication and population 

• Device thinning/delidding 

• Cables, connectors, miscellaneous 

• Test equipment purchase/rental 

• Facility Costs 

- Note that estimating the amount of beam time required is non- 
trivial: modes of operation, ions, temperature, power, etc. all 
factor into the test matrix and need to be prioritized 

• Travel 

• Shipping 
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Device Constraints 


Devices under test (DUTs) can range from very 
simple transistors to the most complex systems 
on a chip (SOC) 

- This range implies test set implementations can vary 
just as widely 

At the top level, the following are the key items to 
begin planning with: 

- Datasheet and 

- Application requirements (mission specific or range for 
“generic” research) 

We note that implementing a test set hinges 
greatly on the DUT type and requirements, 
however, detailed discussion of this is out of 
scope for this talk. 

- Certain key features will be delineated later 
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DUT Parameter Space 

• DUT parameter space may include multiple items found on 
datasheets: 

- Electrical performance 

• Frequency, timing, load, drive, fanout, 10, ... 

- Application capability/ operating modes 

• Processing, configuration, utilization... 

- Power 

- Environmental characteristics, and so on 

• Mission specific testing will limit the space as part of the 
requirements 

- Research tests must consider the overall application space of 
the DUT and determine priorities for configuration of tests 

• We note that device sample size is also considered and may 
be limited due to resource or other constraints. 

- Good statistical methods are still recommended 

- Lot qualification issues should be considered 

• Key features, device markings, etc. should be included 
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Predicting DUT SEE Categories 



* An analysis of the types of SEE the device might 
observe during irradiation is required. 

- This may be called a error/failure mode analysis 

- Predicted type and even frequency of SEEs will drive the 
data capture requirements discussed later as will error 
propagation/visibility 

* An analysis should include 

- Upset (single, multiple, transient, functional interrupts, 
etc..) and destructive issues, as well as, 

- Mission specific objectives (Ex., application 
requirements or destructive test only) 

* Looking at existing data on similar device types 
and technologies may help in this process 
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DUT Data Capture - 
Sample SEU Capture Signatures 

• Upsets can be as simple as a short glitch/transient in an 
output or an incorrect output state 

• Upsets can be complex: 

- Bursts: streaming upsets that are time limited (i.e. occur 
from time x n to x n+k ) 

• Burst vs uncorrectable error? 

- One particle strike may cause an oscillation between 
known good and bad values (metastable) 

• Difficulties 

- Differentiate between a single event versus accumulation: 

• Multiple effects may occur from one particle strike 

• Multiple effects may occur from an accumulation of particle strikes 

- Differentiate between hard errors and soft errors 

• Is it bus contention? 

• Is it a micro-latch? Or... 
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Test Set Requirements 

• Test set requirements are a set of derived requirements from the 
mission/DUT/facility requirements 

- Example: requirement for a test in vacuum may be different than one in 
air 

• Knowing how a DUT performs is one thing, but defining 
requirements for a test system is clearly separate 

- Test set requirements should encompass actual application range or 
have sufficient flexibility such that modifications can be made on site 
easily 

• Mission Requirements generally have ranges of operation. 

- The test set should accommodate this range in areas such as: 

• Min, max, and typical (speed, temperature, voltage) 

• Vary inputs 

• Note the difference between static tests and dynamic tests 

• Output loading 

• We note that a test plan should provide full details, 
schematics, figures, photos, etc. of test method/set 
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Test Set Considerations 

• Test Set Development challenges 

- Visibility of upsets may be restricted with complex devices 

- Testing the expected state of the device may be impossible 

• Test Set considerations 

- May be necessary to separate tests for various portions of the device 

• Example: FPGA (configuration, data paths, and SEFIs) 

- Understand and note test restrictions when determining SEU cross 
sections and error rates 

- Be aware of the separation of tester, user equipment, and DUT during 
testing. 

• Boards for DUTs: roll your own or ??? 

- DUT mounting can be performed by: wiring, soldering, or socketing 

• Wiring will only work for slow devices with minimal I/O count 

• Soldering onto a board will increase the range of angular testing and 
improved speed/noise performance 

• Socketing provides flexibility: if DUT dies, another can easily replace it 

- Potential signal integrity issues must be considered (ground bounce, 
transmission line effects, etc...) 
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Data Requirements 


Data requirements may be broken into two categories 

- Data capture, and, 

- Data analysis 

Data capture, in this context, is not how you capture the 
data, but the requirements/items that should be considered 
for capture 

Data analysis is the other end of the picture: everything 
from the system-wide flow of the data, what format it is 
being captured in, and what are the requirements for 
analyzing this data (real-time and post-testing, as well as 
planning how this should be implemented. 

We suggest treating radiation data much like a spacecraft 
treats science data: a telemetry and command system 

- Utilize as many reliable design practices as possible to have 
confidence in the results 
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Data Capture 

Multiple facets are included in data capture including 

- Data volume and storage 

• Maximum error capture rates should be planned as well in order 
ensure the TBD system can keep up 

- Resolution of measurements 

• This includes “housekeeping” data as well at the “scientific” 
information 

- Timetagging 

- Supply currents 

- Temperature 

- Beam/facility run information, 

- Accumulated dose, and so on... 

- We note that capture criteria per beam run may hinge upon 
beam “stop” criteria 

• X number of errors 

• Beam fluence 

• Current limit 

• Anomaly 

• Other 
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Data Capture - Reliable! 


Some suggested implied requirements for 
reliable data capture 

- Must abide by datasheet requirements (timing 
diagrams, DUT output drive, etc...) 

- Might require the capability to observe short 
duration upsets 

- Should readily capture random errors 

- Should be able to determine changes in current 

- Should be able to keep up with the upset rate by: 

• Storing upset data locally (fastest method - but can be 
restricted by amount of storage) 

• Bandwidth limitations of communications links 

• Some mix of the above two options - alleviates the 
storage and bandwidth issues 

Flexibility to adapt to unexpected “events” 
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Data Analysis 


The early definition of the 
data/command flow and 
structure is key to performing 
a successful test 

- Developing an end-to-end 
data/command flow diagram, 
and, 

- Defining data and command 
packet structure at each point 
along the path 

• Headers (run number, etc...) 

• Word formats and length 

• Insertion of housekeeping 
information 

Note: Geographical (DUT 
layout) and temporal 


Commands: (start, stop, setup) 



END-to-END Data/Command Flow 


information often aid 


determining root cause of error 
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Processing the Data 



* Every plan should include a discussion of how 
the data will be processed whether it’s for 

- Full width half max (FWHM) for transients, 

- Physical mapping of errors and multiple bit events, or 

- Any of the myriad of data events in between. 

* Requirements for what needs to be 
viewed/processed real-time in order to make 
informed decisions at the site as well as what 
should be done as part of post-processing should 
be clearly delineated. 
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Facility Issue - Device Preparation 



M. R. Shaneyfelt, et al. , SEE Symposium, 2011. 


If only everything was hermetic! 

Ion’s range of penetration is short 
compared to packaging materials 

- Cannot use protons for everything 

What is the package style and die 
material? 

- Are there heat sinks? 

Methods: mechanical, chemical, and 
electromagnetic (ablation lasers) 





Open a can 


Acid etch/de-pot plastic encapsulated microcircuits 


SEE Test Planning 101, Seville, SP - LaBel, Pellish, Berg Sep 19 2011 


21 




Facility Considerations - 
Angles and Ion Choice 



^Roll angle 


http://en.wikipedia.org/wiki/Spherical_coordinate_system 


Heavy Ion Facility Comparison 
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Linear Energy Transfer (MeV cm /mg) 


J. A. Pellish, et al., IEEE Trans. Nucl. Sci., vol. 57, 
no. 5, pp. 2948-2954, Oct. 2010. 


• What’s the sensitive area(s) geometry and are there any 
hardening techniques (design and/or process) employed? 

• Is ion range or d£/dx (ionization/length) more important? 


SEE Test Planning 101, Seville, SP - LaBel, Pellish, Berg Sep 19 2011 


22 


Facility Considerations - 

Dosimetry 


• SiGe HBT transistor under 
microbeam irradiation at Sandia 
National Laboratories 

• 36 MeV oxygen 

o Surface LET = 5.3 MeV-cm 2 /mg 

• 60 scans in total 

o Early = first 12 scans 
o Late = last 12 scans 

• Note the large diffusion component 

• Dose/damage from heavy ions 
can be a significant factor 

• Is my device susceptible to this? 



50- 



I 


I 1 1 1 1 T" 

-50 -40 -30 -20 -10 0 


- 1 1 1 1 r ~ 

10 20 30 40 50 


0.88 


O 


0.70 : 



0.53 o. 
O 

0.35 “ 

CO 

CD 

o.is^T 

o 

o.o 


X Position (jjn) 



SEE Test Planning 101, Seville, SP - LaBel, Pellish, Berg Sep 19 2011 


23 


Early Late 


Facility Considerations - 

Dosimetry 


J.-M. Lauenstein, Ph.D. Dissertation, 

U. Maryland, 2011. 

Dose type and bias effects on 
power MOSFET V th 

• Dose/damage from heavy ions 
can be a significant factor 

• Is my device susceptible to this? 
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Early Late 




Facility Considerations - 
Beam Profile and Purity 


1.26 GeV 84 Kr Primary Beam 



SRIM-2008.4 


Degraded Proton Energy Distributions 
14.6 and 63 MeV primaries 



B. D. Sierawski, ef a/., IEEE Trans. Nucl. Sci., 
vol. 56, no. 6, pp. 3085-3092. 


• What is the beam’s emittance (space and momentum)? 

• Where are the sensitive areas on my device under test? 

• How big are the sensitive areas? 

• Am I sensitive to destructive effects? 
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Facility Considerations - 
Beam Profile and Purity 


Low-Energy Proton 
Scattering 


Upstream Ta 
Scattering Foil 

6.5 MeV 
Incident 
Proton Beam 

DUT is ~5 m away 


J. A. Pellish, etai, SEE Symposium, 2011. 


ESA SEU Monitor 



R. H. Sorensen, etai, Proc. RADECS, 2005. 


• What is the beam’s emittance (space and momentum)? 

• Where are the sensitive areas on my device under test? 

• How big are the sensitive areas? 

• Am I sensitive to destructive effects? 
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Staging Area 


Facility Considerations - 
Setup and Cabling 

Texas A&M Cyclotron Facility NASA Space Radiation Lab 


http://cyclotron.tamu.edu/ref/pics/3d_new_reline.png 

http://www.bnl.gov/medical/NASA/CAD/NSRL_Facility_and 

• Is there a staging area? _Target_Room.asp 

• How large is the data collection/user room? 

• What kind of cables/feedthroughs are present? 

• How long is the cable run? (signal bandwidth, voltage droop, etc.) 


Equipment Lift 


Data Room 




- 


Radiation Effects Beam Line 


Labyrinth is over 
30 m long! 


>w 
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Facility Considerations - 
Setup and Cabling 


Avoid 

the 

dreaded 

CABLE CADAVER 


SEE Test Planning 101, Seville, SP - LaBel, Pellish, Berg Sep 19 2011 


28 


Configuration Management (CM) 



• The rule here is simple: know and document what 
you have, what you are using, and how you are 
using it. This ranges from cabling all the way to 
coding! 

- CM defines which version you have and making sure 
you bring the tools to modify if needed 

• Ex., which VHDL code is final one for either the test set or 
DUT (if applicable)? 

• Each team member is responsible for CM 

* Data backup is related 

- Make sure you have a plan for storage of multiple copies 
of the data, who is responsible, and what happens for 
post-processing 
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Logistics 



• While non-technical, logistics related to test 
planning and writing a test plan are no less 
important 

* Areas for consideration in no particular order: 

- Test team member contact info (cell phones, hotels, 
flights, etc...) 

- Facility contact information including maps for newbies 

- Contact information for key people at home site 

- Equipment list including spares 

* Don’t forget datasheets! 

- Shipping/transport of equipment (cost, tracking, ...) 

- Roles and responsibilities of the team 
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Contingency 



* Contingency is required for several reasons: 

- Test set does not work 

- Test set does not work as well as expected 

- Error signatures are different than anticipated 

- Facility may have an “issue” such as the beam goes 
down 

* A good plan will include: 

- Prioritization of tests planned (which devices, which 
tests) 

- Limits on debug time to make a decision to test, move 
to a later test timeslot, or ??? 

* Example: if after 1 .5 hours no significant progress is 
noted, go to backup device 

- Backup devices (in case test ends early or other 
device/test doesn’t work properly) 
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Summary 


SEE Test Plan Outline - 

Introduction and objectives 
Detailed Device Information 
Documentation 

- Block diagrams, circuit diagrams, cabling diagrams, 
datasheets, etc... 

- Photos of device and test set 
Equipment list 

- Packing and shipping information (detailed) 

Test Methodology and Data Capture 

- Including Data Storage Structure 
Configuration management 

- Data backup and distribution plan 
Personnel and Logistics 

Data Analysis Plan 
Contingency Plan 
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Summary 



* This section of the short course was designed to 
provide the user the basic thought processes 
required to develop a successful test plan 

- Technical issues, 

- Logistics issues, and, 

- Programmatic issues. 

* Further details are found in the full notes 
accompanying this presentation. 
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